
Online credit recovery: 
Enrollment and passing 
patterns in Montana 
Digital Academy courses 



This study of 2013/14 Montana Digital Academy (MTDA) online credit recovery 
courses found that: 

More male students than female students enrolled in MTDA online credit 
recovery courses, and students in grades 10 and 11 accounted for a larger 
proportion of MTDA student enrollments than those in grades 9 and 12. 
Enrollments in MTDA online credit recovery courses were higher in English 
language arts than in any other subject. 

The passing rate in MTDA online credit recovery courses was 57 percent 
of student enrollments. 

Passing rates in MTDA online credit recovery courses were lowest in math 
(49 percent) and English language arts (52 percent). 

Passing rates were lower for students who took one MTDA online credit 
recovery course per semester (40 percent of enrollments) than for students 
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Summary 


Most U.S. school districts (88 percent) offer credit recovery courses or programs for 
students. In rural states such as Montana, online credit recovery options are especially 
popular because they allow schools to serve students in remote areas throughout the year, 
across a range of subjects, and with few additional resources (Carver & Lewis, 2011; Oliver, 
Osborne, Patel, & Kleiman, 2009). Such programs offer students greater flexibility and 
choice, which results in more opportunities to make up classes and a greater likelihood 
that students will stay in school and stay on track to graduate (Oliver et al, 2009). 

Despite the growing popularity of online credit recovery courses, there is little research 
about which students take these courses or how well they perform. This report address- 
es this information gap by examining 2013/14 data from the Montana Digital Academy 
(MTDA), the only state-funded program offering online credit recovery courses in 
Montana. The report provides a descriptive analysis of course enrollment and passing 
rates. It also draws on interviews with eight district leaders and one MTDA representa- 
tive to provide context for the analysis and to describe other credit recovery strategies in 
Montana. The report offers educators an early look at the potential of online credit recov- 
ery courses to help struggling students get back on track to graduation. It can also help 
state leaders compare the MTDA with other online programs and identify possible areas 
for investigation or improvement when designing credit recovery options. 

The report’s key findings are: 

• More male students than female students enrolled in MTDA online credit recov- 
ery courses, and students in grades 10 and 11 made up a larger proportion of 
MTDA student enrollments than those in grades 9 and 12. 

• Enrollments in MTDA online credit recovery courses were higher in English lan- 
guage arts than in any other subject. 

• The passing rate in MTDA online credit recovery courses was 57 percent of 
enrollments. 

• Passing rates in MTDA online credit recovery courses were lowest in math 
(49 percent) and English language arts (52 percent). 

• Passing rates were lower for students who took one MTDA online credit recovery 
course per semester (40 percent) than for students who took multiple courses in a 
semester (nearly 80 percent). 
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Why this study? 


The social and economic costs of not completing high school are enormous. Students who 
drop out of high school earn less over their lifetime than graduates and are more likely to 
be unemployed (Day & Newburger, 2002; Heckman & LaFontaine, 2007; U.S. Bureau of 
Labor Statistics, 2015). Students who drop out also experience more health problems, have 
a shorter life expectancy, and are at a higher risk for incarceration (Muennig, 2005; Sum, 
Khatiwada, & McLaughlin, 2009). High school dropouts from a single graduation cohort 
can represent $150 billion in lost wages (Alliance for Excellent Education, 2011; Rouse, 
2005). At current graduation rates that would result in economic losses totaling $1.5 trillion 
over the next decade (Alliance for Excellent Education, 2011). The country also spends 
billions of dollars in additional public funds on health care, public assistance, and other 
services for adults without high school diplomas (Levin, Belfield, Muennig, & Rouse, 2007). 

Staying on track for graduation depends heavily on the continual accrual of required 
course credits. Students who fall behind in earning credits toward graduation are at a high 
risk of leaving school prior to graduation (Allensworth & Easton, 2007; Bruce, Bridgeland, 
Fox, & Balfanz, 2011). A student who fails just two semester courses during grade 9 has 
little more than a 50 percent chance of graduating in four years; a student who passes all 
his or her courses has an 85 percent chance of graduating (Allensworth & Easton, 2007). 


Despite the 
growing use of 
online credit 
recovery programs, 
research on 
their efficacy 
remains limited 


Given the importance of credit accrual, state and local leaders across the country have 
turned to credit recovery courses as a strategy to prevent high school dropouts and improve 
graduation rates (Greaves & Hayes, 2008; Picciano & Seaman, 2009). Credit recovery pro- 
grams allow students to retake courses they failed. As of 2011, 88 percent of U.S. school 
districts offered some type of credit recovery program or course that employs various 
approaches, including alternative school models, extended learning time programs (for 
example, after school), traditional face-to-face summer school, and online learning options 
(Carver & Lewis, 2011; Berliner, Barrat, Fong, & Shirk, 2008). 


States and districts with large rural populations find online credit recovery options to be 
particularly attractive because they allow schools to serve students in remote areas through- 
out the year, across a range of subjects, and with few additional resources, while offering 
greater flexibility and choice for students (Oliver, Osborne, Patel, & Kleiman, 2009). States 
with large rural populations, such as Kentucky, Michigan, Missouri, and Montana, have 
recently invested in online credit recovery programs for struggling students. 

Despite the growing use of online credit recovery programs (Picciano, Seaman, Shea, & 
Swan, 2012), research on their efficacy remains limited. The research on student perfor- 
mance in online classes has generally found passing rates of 30-60 percent (Blazer, 2009). 
Early, small-scale studies suggest that online credit recovery may boost graduation rates 
(Franco & Patel, 2011; Mileaf, Paul, Rukobo, & Zyko, 2012). Yet, there are many gaps 
in what is known about student performance in these classes. For example, very little is 
known about who enrolls in online credit recovery courses, the rate at which students 
complete a course and receive a passing grade, or how enrollment and passing rates vary by 
student subgroups, subject area, or the time of year when courses are taken. 

This report looks specifically at 2013/14 course enrollment and passing rate patterns in the 
Montana Digital Academy (MTDA), a statewide program that offers online credit recovery 
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courses. Although the report also describes other credit recovery strategies in Montana, 
it focuses on the potential of online credit recovery courses to help struggling students 
get back on track to graduation. The findings can help state leaders compare the MTDA 
with other online programs and identify areas for additional investigation or improvement 
when designing credit recovery options. 


Credit recovery in Montana 


The Montana Office of Public Instruction launched the Graduation Matters Montana 
initiative in 2010 to encourage schools and communities across the state to focus on 
keeping students on track to graduate from high school. Graduation Matters Montana 
supports communities that implement local initiatives to address high school graduation 
and postsecondary readiness. Since the start of Graduation Matters Montana, the state’s 
high school dropout rate 1 decreased from 4.3 percent in 2010/11 to 3.4 percent in 2014/15, 
and the graduation rate rose from 82.2 percent to 85.4 percent (Montana Office of Public 
Instruction, Graduation Matters Montana, 2015). 

Many Graduation Matters Montana districts include credit recovery programs, defined 
as courses or coursework for students who have failed a class, as a strategy for improving 
graduation rates. What constitutes failure can vary from district to district, but generally 
indicates end-ofiyear course grades below 60 percent. Many districts in Montana also have 
a course attendance requirement, which results in automatic course failure if a student has 
more than 10 unexcused absences. 


The report focuses 
on the potential 
of online credit 
recovery courses 
to help struggling 
students get 
back on track 
to graduation 


“Rural states face unique challenges in implementing federal reforms and overcoming geo- 
graphic and economic barriers,” according to Montana Superintendent of Public Instruc- 
tion Denise Juneau (Hagen, 2011). These challenges can include lack of access to a large 
pool of certified teachers or expansive community-based supports and a growing student 
population. Nationally, growth in rural enrollment is outpacing growth in nonrural 
enrollment, with rural schools now serving more than 23 percent of public school stu- 
dents (Strange, Johnson, Showalter, & Klein, 2012). In Montana, school and district staff 
members said that some employees commute one to two hours each way, making it difficult 
to provide student services outside of regular school hours. Given this context, demand for 
online credit recovery options in Montana is projected to grow steadily (Miller, 2012). 

The Montana Digital Academy 

The MTDA is the only state-funded provider of online courses used by Montana public 
schools. Hosted at the University of Montana, the MTDA offers online courses for credit 
recovery and original credit. The MTDA Connect Credit Recovery Program offers more 
than 50 options for credit recovery to high school students in core subjects such as English 
language arts, math, social studies, and science, as well as a limited number of elective 
courses. The MTDA aligned its curriculum with the Common Core State Standards in 
English language arts and math after Montana officially adopted the standards in fall 2011, 
according to the MTDA administrator (see box 1 for a description of how the MTDA 
program has changed over time). 

Most MTDA credit recovery classes have three to six units, with pretests that allow stu- 
dents to test out of individual modules in each unit. The program blends MTDA staff 
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Box 1. Changes to the Montana Digital Academy structure over time 


The Montana Digital Academy (MTDA) credit recovery program has evolved based on feed- 
back from educators and students since it began offering classes in 2011. One change in the 
2012/13 school year was to allow open enrollment throughout the year. Districts using the 
MTDA reported a need for year-around enrollment for a number of reasons, including credit- 
deficit students transferring into the district after the enrollment period closed and students 
whose grades were too low after six weeks in their traditional course to be able to catch up 
and pass by the end of the semester. 

Another change in the 2012/13 school year was to limit the number of students who 
can enroll in more than one course at a time. Schools wanting to enroll students in multiple 
courses simultaneously must lobby for students to do so by proving that the district can supply 
the student with sufficient time and support to complete the courses. This change resulted 
from an internal MTDA review of course passing rate data that suggested that students who 
took multiple classes simultaneously were less likely to finish any of them. 


(certified teachers who serve as academic area coaches) and local school personnel to help 
students through the course. (See box 2 for a description of other credit recovery options 
in Montana.) 

Although it is not mandatory, the MTDA encourages districts to provide a supportive 
environment to students taking online courses (Montana Digital Academy, n.d.). 2 The 
MTDA describes a supportive environment as a local site that: 

• Addresses technical issues such as enrollment, setting up a username and pass- 
word, logging in, finding the course guide, accessing the materials, and assisting 
students with access and facilities. 

• Addresses academic issues such as monitoring progress, engaging slow-paced or 
unresponsive students, and helping students access local resources (for example, 
tutors, one-on-one assistance). 

• Ensures that students have daily access to a computer with high-speed Internet 
service at home and before, during, and after school. 

• Ensures that students have regularly scheduled access to school-based mentoring/ 
counseling services, strong parental support, and use of best practices to support 
them as online learners as identified in district policies. 

• Offers on-site proctoring of student exams. 


Montana Digital 
Academy offers 
more than 50 
options for credit 
recovery to high 
school students 
in core subjects, 
as well as a 
limited number of 
elective courses 


This support can range from helping fill out entrance counseling forms and reviewing 
transcripts to determining proper course placement and ensuring reliable computer access. 
The MTDA also requests that districts provide adult monitoring and oversight at the 
beginning of the course and at least weekly throughout the course, as well as proctoring for 
unit and semester finals. 


What the study examined 


This study responds to a request from Montana district and state education leaders for 
information on enrollment and course outcomes in online credit recovery classes offered 
through the MTDA. It builds on an earlier report (Miller, 2012) that used data from 
the initial implementation of the MTDA in the 2011/12 school year. This Regional 
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Box 2. Proactive strategies to support students from failing courses in Montana 


Interviews with district leaders showed that in addition to using online recovery courses through 
the Montana Digital Academy, districts also use such online programs as Acellus, Odyssey 
Ware, and Compass. In addition, schools offer face-to-face recovery options during nonschool 
hours, such as after school, on Saturday, evenings, and during school breaks. Students can 
also repeat courses during the regular school year at their own school or at an alternative high 
school. 

Some districts have begun to think about how to support more students before they fail a 
course and need credit recovery. One strategy that Montana districts have found promising is 
catch-up sessions during breaks in the school year — often referred to as “just in time” credit 
recovery. Districts across the country are using this proactive model with greater frequency 
(Mileaf et al., 2012). With this option, teachers and administrators continuously monitor stu- 
dents’ progress and provide extra support as needed. 

One school district leader reported that during a few days over winter break certified teach- 
ers administer course content tailored to fill each referred student’s gap in knowledge. The 
initiative requires collaboration and buy-in from all classroom teachers, who must identify what 
the student needs and prepare a concise set of materials that allow students to demonstrate 
their understanding of the course content. Teachers who proctor the catch-up sessions are 
compensated. 

“The catch-up sessions used to span five days over [winter] break, but we found that we 
lost the kids [at] about day three,” said one district assistant superintendent. “Now we try to 
focus more on the specific content by asking the teachers what exactly the kids need to know 
to move on to the next content area.” 

This district expanded the program by offering catch-up days periodically throughout the 
year. The catch-up days serve as an incentive to all students. Those who are missing work 
and struggling in their courses must come to school on these days to make up missed assign- 
ments, while students who are doing well get the day off. Some Montana districts also find 
early warning system indicators — designed to flag students when they first begin to struggle 
— useful for proactively tracking students in need of credit recovery (Frazelle & Nagel, 2015). 

Other proactive examples of credit recovery include targeting course accrual for students 
in grades 9 and 10. As one administrator explained, “If a student is missing an English lan- 
guage arts credit at the end of their freshmen year and we require four English credits to grad- 
uate, we know they’re going to need to make up a credit eventually.” 

Some districts are eliminating or altering the 10-day absence policy, which states that 
students cannot receive credit for a course if they have 10 or more absences, even if they 
demonstrate mastery of the content. Several district administrators said that this policy can 
be a barrier as they strive to make the credit accrual process less reactive and more proactive. 
These administrators offered a critique of the conventional wisdom that students must have 
near-perfect attendance and complete every assignment to make the grade. One administrator 
described the effort to move beyond this mindset: “Some school staff members were resistant 
to credit recovery because they questioned the rigor or fairness of the credit recovery practic- 
es. It took about a year of meetings and discussions to move on from that and get everyone 
on board. [You have to] consider your own struggles or step into the shoes of these students. 
What if you got sick or your parents got divorced and you became depressed? Those are the 
kinds of students the credit recovery program is designed [for]. Additionally, teachers taking 
ownership of their students’ success was essential." 
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Educational Laboratory Northwest study examines data from the 2013/14 school year and 
describes how enrollment patterns and passing rates have changed since the first year of 
the program. All data are reported as total enrollments, not individual students, because a 
student may enroll in multiple classes. The report addresses five questions: 

• Who enrolls in the MTDA, by gender and grade level? 

• How is student enrollment distributed across subject areas? 

• What are the MTDA course passing rates, and how do they vary by gender and grade? 

• How do passing rates vary across subject areas? 

• How do passing rates vary based on the number of courses a student takes? 

This study draws on data provided by the MTDA, as well as interviews with one representa- 
tive from the MTDA and eight school district leaders across the state to provide context for 
the descriptive analysis. Because Montana stakeholders were also interested in learning about 
other credit recovery strategies across the state, the interviews addressed this topic as well (see 
box 2). Details on the data sources and methodology for this study are in appendix A. 

What the study found 


The MTDA’s overall passing rate is 57 percent. This falls at the high end of the range 
found in prior studies of online classes, according to Blazer (2009), indicating that stu- 
dents in the MTDA online credit recovery programs do at least as well as students in 
other online courses. The following section describes detailed findings on enrollment and 
passing rates in MTDA online credit recovery courses. 

More male students enrolled in Montana Digital Academy online credit recovery courses than 
female students, and students in grades 10 and 11 make up a larger proportion of student 
enrollment than those in grades 9 and 12 

A total of 2,452 students accounted for 3,763 unique enrollments in MTDA online credit 
recovery courses during the 2013/14 school year. Sixty percent of MTDA enrollments were 
male students, 34 percent were in grade 10, and 31 percent were in grade 11. Only 19 percent 
of enrollments were in grade 9. The remaining 16 percent were in grade 12 (figure 1). 

More students enrolled in English language arts Montana Digital Academy online credit recovery 
courses than in any other subject area 

English language arts credit recovery courses accounted for the largest share of MTDA 
enrollments (37 percent; figure 2). Enrollments in the remaining courses were fairly evenly 
divided among math (19 percent), social studies (19 percent), and science (17 percent). 
These course enrollment patterns mirror those in the MTDA’s first year of implementa- 
tion, in 2011/12. Of the 3,409 credit recovery course enrollments in 2011/12, 38 percent 
took an English language arts class, 22 took social studies, 18 percent took math, and 
15 percent took science (Miller, 2012, p. 14). 

Fifty-seven percent of Montana Digital Academy online course enrollments resulted in a passing 
grade, representing a 12 percentage point drop in passing rates compared with 2011/12 

The MTDA defines a passing grade as an end-oficourse score of 60 percent or better. 
Of students who did not receive a passing grade, all but five dropped out of their course 


The Montana 
Digital Academy’s 
overall passing 
rate is 57 percent, 
which falls at the 
high end of the 
range found in 
prior studies of 
online classes 
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Figure 1. More male students and grade 10 and 11 students enrolled in Montana 
Digital Academy credit recovery courses, 2013/14 
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n = 3,763 unique course enrollments. 




Source: Authors' analysis of 2013/14 Montana Digital Academy enrollment data. 


Figure 2. English language arts had the highest enrollment of Montana Digital 
Academy credit recovery courses by subject, 2013/14 
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n = 3,763 unique course enrollments. 

Source: Authors' analysis of 2013/14 Montana Digital Academy enrollment data. 
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before receiving a grade. 3 Passing rates were fairly consistent across gender and grade 
levels, although female students (60 percent) and grade 12 students (63 percent) performed 
slightly better than other student subgroups (figure 3). In 2013/14, 57 percent of students 
enrolled in an MTDA online credit recovery course received a passing score, compared 
with 69 percent in 2011/12 — a 12 percentage point drop (Miller, 2012). 


Passing rates in Montana Digital Academy online credit recovery courses were lowest in math and 
English language arts classes 


Passing rates were just 49 percent for students attempting to recover math credits in 2013/14 
(figure 4). Despite the high demand to recover English language arts credits, passing rates 
were similarly low in these courses (52 percent). Social studies courses had the highest 
passing rates of all classes (71 percent). 

Students taking one Montana Digital Academy online credit recovery course per semester had lower 
passing rates than those taking multiple courses in a semester 

Among students taking only one MTDA online credit recovery course in a semester, 
60 percent ultimately failed or dropped the course. However, the 21 percent of students 
(570 students) who simultaneously took multiple courses in at least one semester had con- 
sistently higher passing rates (figure 5). For students taking multiple MTDA credit recovery 
courses in any semester, the passing rate was 68 percent for students who enrolled in two 
courses, 82 percent for students who enrolled in three courses, and 85 percent for students 
who enrolled in four courses. Only a few students took more than four courses. This trend 
was consistent across the fall 2013, spring 2014, and summer 2014 semesters (see box 3 for 
a discussion of challenges that may affect MTDA passing rates). 4 The MTDA requires 
schools to provide extra time and supports for multiple course takers (see box 1). 


English language 
arts credit 
recovery courses 
accounted for 
the largest share 
of enrollments; 
enrollments in the 
remaining courses 
were fairly evenly 
divided among 
math (19 percent), 
social studies 
(19 percent), 
and science 
(17 percent) 


Figure 3. Passing rates for Montana Digital Academy recovery courses were higher 
for female students and for grade 12 students than for other subgroups, 2013/14 
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n = 3,763 unique course enrollments. 

Note: Course passing is defined as students earning a grade of 60 percent or higher. 
Source: Authors' analysis of 2013/14 Montana Digital Academy enrollment data. 
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Figure 4. Student passing rates were highest in social studies courses and lowest 
in English language arts and math courses, 2013/14 
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Note: Course passing is defined as students earning a grade of 60 percent or higher. 
Source: Authors' analysis of 2013/14 Montana Digital Academy enrollment data. 


Figure 5. The aggregate percentage of students passing Montana Digital Academy 
online credit recovery courses was higher for students enrolled in multiple courses 
in one semester, 2013/14 
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Note: n values are the number of unique course enrollments. Course passing is defined as students earning 
a grade of 60 percent or higher. Calculations of passing rates for multiple enrolled courses are based on the 
total number of students taking multiple courses in any one semester, regardless of time of year. Since the 
same student has the potential to take multiple courses in each semester (fall, spring, and summer), the 
total number of students exercising this option is more than the number of unique students who enrolled in a 
course over the 2013/14 school year. 

Source: Authors' analysis of 2013/14 Montana Digital Academy enrollment data. 
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Box 3. Challenges to passing Montana Digital Academy online courses 

Interviews with school and district leaders charged with administering credit recovery programs 
in Montana suggested several factors that may undermine students’ ability to pass MTDA 
credit recovery courses. First, students taking credit recovery courses often display low levels 
of engagement, such as not regularly attending school, which may have contributed to their 
initial course failure. “We’re still faced with the challenge of attendance," one leader observed. 
“The kids have not always had good attendance, and it can be hard to change that.” Another 
coordinator explained that poor attendance can stand in the way of accruing recovery credits 
because of the 10-day absence policy many districts employ: “Some kids are right down to the 
wire — if they could come for one full day they would get credit, and they don’t show up.” Thus, 
absences and other forms of disengagement may be serious barriers to recovering needed 
credits. 

Second, many schools using the MTDA do not provide extensive support to help students 
pass online courses. Most interviewed educators reported that supports took the form mainly 
of assisting with technological issues and providing computers for students to use for online 
classes. They reported that academic support was generally limited to tracking students’ prog- 
ress through the curriculum and providing updates on the coursework students still need to 
complete. For example, one educator said that while monitoring classes, “I’ll remind them of 
the progress they’re making. I can not only track their weekly progress in a class, but also 
how many classes they’ve completed, how long it’s taken them, and so on. So, if someone is 
a sophomore or junior I talk to them about their pace — ‘You’re never going to make it at this 
rate.’” 

While a few interviewees said they do provide support to students after they fail a quiz or 
test, most stated that they considered the classroom instructors’ primary role as managing 
the online process. The MTDA provides a list of recommended student supports (appendix B), 
but districts are not required to provide them unless students are simultaneously enrolled in 
multiple courses. 

Finally, the academic difficulty of the courses appears to have increased. When describing 
the impact of aligning the online credit recovery courses with Common Core State Standards, 
the assistant director/curriculum director for the MTDA stated: “Generally speaking, [based on] 
internal evaluations with our teachers, all the courses did take a pretty heavy step forward in 
rigor. Typically, in the English language arts courses, we’re requiring a lot more deep reading, 
which is similar to the Common Core. Of course, the challenge is that in a lot of cases stu- 
dents are taking these as make-up courses, and the original course at their local school may 
or may not have been aligned to the Common Core." 

District administrators confirmed this challenge, stating that some students have strug- 
gled with the increased academic rigor of the online credit recovery courses, especially when 
courses introduced concepts or approaches that students had not been exposed to in the 
traditional classroom. 


Implications of the study findings 


As online credit recovery programs across the country continue to expand, stakeholders 
will need more information on student success in these programs compared with other 
credit recovery options, as well as clear guidance on how to implement these programs 
successfully. The findings in this report present several implications for district and state 
leaders in Montana and potentially elsewhere to consider. 
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Schools may want to monitor when students are attempting to recover credits and in which courses 

Failing a course in grade 9 is one of the strongest predictors of not graduating from high 
school (Allensworth & Easton, 2007). Research has also shown that many students expe- 
rience academic challenges as they transition from middle school to high school (Benner 
& Graham, 2009; Rosenkranz, de la Torre, Stevens, & Allensworth, 2014; Simmons & 
Blyth, 1987). Given that only 19 percent of students who enrolled in MTDA credit recov- 
ery courses were in grade 9 and that other studies have documented that about half of 
grade 9 students fail at least one course (see, for example, Allensworth & Easton, 2007), it 
is possible that students may be waiting to recover credits later in high school, when the 
consequences of unsuccessful recovery attempts put them at greater risk of not graduating. 

In subjects that require four credits to graduate (such as English language arts for Montana 
schools), monitoring credit accrual rates could be crucial for increasing on-time graduation 
rates. While online course enrollments from the MTDA provide one picture of the types 
of courses students across the state need to recover, districts should examine their own 
data to decipher which courses the majority of their students are struggling with. Schools 
might consider how to help students prevent or recover failed classes immediately and stay 
on track for graduation (see box 2 for a discussion of proactive credit recovery strategies 
used by Montana administrators). Districts implementing early warning systems often use 
credit accrual as an indicator of whether a student is on track for graduating in four years 
(Frazelle & Nagel, 2015). Combining an early warning system that flags grade 9 students 
who fail a course with access to a credit recovery option could keep more students on track 
for graduating with their cohort. 

Montana Digital Academy may want to examine potential differences in the structure of courses 
with higher completion rates 

Online courses in social studies, business, and health/physical education had the highest 
completion rates. The MTDA may want to examine the content delivery structures used 
in these courses and compare them with those used in English language arts and math — 
the courses with the lowest completion rates. Potential differences could be explored to 
see whether the way content is provided and tested in the courses with the highest passing 
rates makes it easier for students to learn in an online environment. 

If the difference in passing rates stems from an increased rigor of courses aligned with the 
Common Core State Standards, as interviews with MTDA and district leaders suggest (see 
box 3), a broader discussion of the role of credit recovery may be called for, particularly 
as original course credit content offered in the districts is still transitioning to meet new 
content standards. 

Students taking Montana Digital Academy online credit recovery courses may benefit from 
additional supports from schools and districts 

According to interviews with district leaders, students taking MTDA online credit recov- 
ery courses sometimes engage in poor academic behaviors, such as not regularly attending 
school or not completing their coursework on time. These types of behaviors put credit 
recovery students at a higher risk of failure, regardless of the format of the course. In addi- 
tion, the MTDA’s efforts to align credit recovery courses to more demanding Common 


Online courses 
in social studies, 
business, and 
health/physical 
education had 
the highest 
completion rates 
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Core State Standards may increase the likelihood of failure, since increasing academic 
rigor often leads to declining grades (Allensworth, Gwynne, Pareja, Sebastian, & Stevens, 
2014). Thus, additional supports from schools and districts may be important for credit 
recovery students. These supports may be especially beneficial for students taking their 
first course in an online environment or in English language arts and math, which have 
relatively low passing rates. 


The finding that students had higher completion rates when enrolled in multiple MTDA 
courses in a semester may highlight the potential positive impact of additional supports. 
The assistant director/curriculum director of the MTDA Connect program explained that 
districts that request that a student be allowed to take more than one course in a semester 
are required to show that they can provide a supportive environment or supply the student 
with more than one scheduled period in which to complete the courses. Although this 
study did not examine whether additional supports contribute significantly to increased 
completion rates, that conclusion is consistent with findings in the current literature. 
Other studies suggest that the benefits of online credit recovery may not be fully realized 
without supplemental academic and emotional supports for students, as well as technical 
supports (Archambault et al, 2010; Bakia et al., 2013; Pettyjohn, Kennedy, & LaFrance, 
2013). 

The higher passing rates of multiple-course takers may also be due to a selection effect. 
Since schools need to justify why students should be allowed to take multiple courses in 
a semester, students who receive this endorsement may have shown the ability to handle 
increased academic challenge in the past and therefore are more likely to pass recovery 
courses. Alternatively, it is reasonable to assume that students who need to recover muh 
tiple courses struggle with schoolwork in general, since they have failed more than one 
course, and therefore may have more difficulty completing multiple courses than students 
who need to make up only one course. In addition, students taking more than one online 
credit recovery course in a semester would typically have less time to study for each class if 
their class schedules were not adapted to accommodate the additional course load. Thus, 
these descriptive findings should be interpreted with caution until future research teases 
out possible explanations. 
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from schools and 
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Limitations of the study 


While providing useful descriptive information, this study was unable to address many 
questions because of restrictions in available data. For instance, because data on overall 
course failures in Montana were unavailable, the study did not calculate the percentage of 
students who failed a class and subsequently enrolled in the MTDA program. Therefore, 
it is unknown how many Montana students need to recover credits but are not using the 
program. 

Additionally, enrollment and passing rate data for credit recovery options outside of the 
MTDA were not available, making it impossible to compare the effectiveness of the 
MTDA with other programs offered in Montana. Thus, the findings are not generalizable 
to other online credit recovery programs. The study did not examine completion rates 
by student subgroups such as race/ethnicity, special education status, and English learner 
status, because the small sample sizes could make it possible to identify individual students. 
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Finally, information provided by interview respondents may not be representative of views 
more broadly held by districts across Montana because of the relatively small number of 
interviews. For example, the proactive strategies presented in this report may not cause 
fewer students to fail courses. Rather, readers may use these strategies as a discussion piece 
when considering their own systematic approach to credit recovery. 
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Appendix A. Study methodology 


This appendix describes the data sources and study methodology. 

Description of qualitative data 

To provide context for the results of the descriptive analysis of Montana Digital Academy 
(MTDA) data, the study team interviewed representatives from the MTDA and eight 
Montana school districts (table Al). 

Participants were asked to describe the challenges they faced in implementing credit 
recovery programs, including challenges related to diagnostic testing, tracking student par- 
ticipation in credit recovery, and tracking student progress in the courses. The interview 
questions were modeled on a survey from the New York Comprehensive Center, which pro- 
vided a useful way of looking at credit recovery implementation through delivery methods, 
program characteristics, staff roles, and the successful recovery of course credits (Mileaf 
et al., 2012). However, that instrument was intended for interviewing vendors, not school 
or district staff members implementing credit recovery programs. Therefore, the interview 
protocol for this study was based on a subset of the New York Comprehensive Center 
questions supplemented with original questions to elicit contextual information about the 
challenges the Montana districts face. (See appendix C for the interview protocol.) 

Qualitative data sample 

The qualitative data for this report are based on nine interviews with one representative 
from the MTDA and one from each of eight Montana school districts. While it is beyond the 
scope of this study to provide complete documentation of credit recovery efforts across the 
state, the study team used purposeful sampling in selecting the districts to reflect the breadth 
of Montana districts and the different credit recovery programs offered. Factors considered 
in selecting the districts included the credit recovery programs offered and the district size 
and location (for example, both large urban districts and small rural districts were included) 
so that the study would represent the widest possible variety of districts. By selecting the 
sample purposefully, the study team was able to avoid selecting districts without replicable 
credit recovery programs or oversampling districts that offer the same program. This method 
gives voice to the diversity of districts in Montana, which includes a large number of Native 
American reservation districts in addition to rural and urban locations. 


Table Al. Description of districts with interviewees 


1 District 

Enrollment 

Region of state 

Urban/rural 

A 

1,090 

Four Rivers 

Rural 

B 

467 

Western 

On/near reservation 

C 

1,966 

North Central 

On/near reservation 

D 

1,808 

South Central 

On/near reservation 

E 

8,078 

Four Rivers 

Urban 

F 

5,778 

Northwest 

Urban 

G 

841 

Northeast 

On/near reservation 

H 

397 

Hi-Line 

On/near reservation 

Source: Montana Office of Public Instruction. 
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The study team worked with staff members from Montana’s Office of Public Instruction 
to select the eight school districts included in the study. These staff members collaborate 
closely with educators across the state to support districts’ activities through the Gradu- 
ation Matters Montana initiative and were therefore familiar with the contexts of these 
districts. Additionally, Graduation Matters Montana provides opportunities for districts 
to share the broad concepts of their initiatives. This type of information helped Office of 
Public Instruction staff members identify a diverse group of districts, by size, location, and 
type of program offered. Once the potential sample was identified, the study team contact 
ed the selected districts to discuss their participation in the study. 

Description of quantitative data 

This study is based on MTDA enrollment data for the 2013/14 school year for students who 
enrolled in online credit recovery courses. Only the latest full year of data was used for two 
reasons: to describe enrollment in online credit recovery programs as they are currently 
delivered and to avoid pooling data across multiple years because online credit recovery 
course offerings or program policies may have changed over that time, making it difficult 
to make inferences from the full dataset or to make year-to-year comparisons. 

The MTDA provided the following student data for the fall, spring, and summer terms of 
the 2013/14 school year: 

• Student identifier. 

• District identifier (for enrollment data from the MTDA). 

• Student grade level. 

• Academic subject area (for example, math). 

• Student’s grade in the course. 

Data collection 

Through a data-sharing agreement with Regional Educational Laboratory Northwest, the 
MTDA shared student data in a secure manner that complies with the Family Educational 
Rights and Privacy Act (FERPA). To protect the privacy of the districts and their represen- 
tatives who participated in interviews, district names, individuals’ names, and the names of 
specific credit recovery programs are not included in the report. To protect student privacy, 
the MTDA agreed to uphold FERPA compliance as a precondition for participating in the 
enrollment analysis. 
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Appendix B. Local management 


This appendix includes a diagram from the credit recovery handbook that the Montana 
Digital Academy (MTDA) provides to districts considering whether to participate (figure 
Bl). The diagram shows a list of supports that the MTDA commits to providing, as well 
as expectations for supports at local sites. Note that sites are not required to provide these 
supports unless their district requests that students take more than one course at a time. 
Other districts and schools considering online credit recovery might consider how their 
supports overlap with the list of expectations for local sites. 


Figure Bl. Supports recommended for local sites and Montana Digital Academy 
supports for students taking online recovery courses 

Local Management of MTDA Connect Credit Recovery 

Due to the population typically served by credit recovery programs, MTDA builds its program based on 
the assumption that the local school and MTDA are partnered to offer students service. Based on our 
history of enrollment, students who are enrolled without local support are highly unlikely to complete a 
credit recovery course. Although each successful program is different, the program setup that brings the 
most success is typically: 


Local site 

Enrollment 

Username and password assistance , 


MTDA 

• Digital curriculum and platform 

• Helpdesk services and technical 


and management for students 

/ \ support \ 

/ • Assistance in getting started 

/ \ • Monitoring of student progress \ 

(logging in, finding the course 

\ • Assistance with specific lesson or \ 

guide, accessing materials) 


topic trouble (may not be available in 

• Monitoring progress and 

Student 

real time; local schools should plan to 

engaging slow-paced or 


supplement) 

unresponsive students 


• Assessment of non-objective 

• Proctoring all unit and semester 


assignments / 

\ exams \ / 

• Evaluation of course completion / 


Helping students access local 
assistance resources (tutors, 
one-on-one assistance, etc.) 
Assisting students with access ar 
facilities 



Recommendation of credit 



Source: Montana Digital Academy’s credit recovery handbook provided via email. 
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Appendix C. Interview protocol 


This appendix shows the interview protocol used to collect information about credit 

recovery options and implementation challenges. 

Program background 

1. Why did you choose to offer each type of credit recovery program? Please describe any 
innovative features of your programs. 

2. Based on your observations, is the need for credit recovery higher at certain student 
grade levels? Why do you think that is? 

3. Based on your observations, do you think more or fewer students need to recover credit 
compared to credit deficiencies in the past? Why do you think that is? 

4. What kind of training, if any, do you provide for staff about facilitating these credit 
recovery programs? 

5. What are the provisions for students with disabilities? 

6. What different courses or accommodations are offered for English learner students? 

7. If you do not track student outcomes, what data are you missing or what supports 
would you need to measure student success? 

Course information 

1. Please provide a list of the specific courses for which you offer credit recovery (for 
example, first semester geometry). (Probe, if applicable: for which courses do you offer 
diagnostic pre-tests and allow students to test out of sections of the course?) 

2. Based on your observations, are there certain subject areas or certain specific courses 
for which more students need to recover credit? Why do you think that is? 

3. What roles do local staff and remote instructors play for online programs? 

4- For online courses: What technical and instructional supports are available for students 
(for example, help lines for website or software questions, help lines or in-person staff 
support for course content questions, resources such as online dictionaries or calculators)? 

5. How are student participation and progress tracked throughout the course? 

Challenges 

1. What are some of the challenges the program has faced (for example, start-up difficul- 
ties, resource constraints, staff and/or student buy-in, implementation, diagnostic testing, 
tracking student participation and progress)? How have you overcome these challenges? 
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2. What supports would help you overcome the challenges you still face? 

3. What advice would you give to other districts that want to provide credit recovery 
options for their students? 
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Notes 


1. The Montana high school dropout rate counts students who were enrolled in school 
on the date of the previous year’s October enrollment count or at some time during 
the previous school year and were not enrolled on the date of the current school year’s 
October count. 

2. Districts are not required to provide these supports unless a student is taking more 
than one course at a time. In addition, district supports provided to a student taking 
more than one course may not be applied to all students taking an MTDA credit 
recovery course in the district. 

3. According to an MTDA representative, students may receive a “course dropped” status 
for one of four reasons: 2-3 weeks of inactivity, followed by warnings; the student 
transferred to another school, dropped out, or moved to another state; the student was 
misenrolled; or the student did not complete the coursework and the academic coach 
requested that the student be dropped rather than fail the course. 

4. Possible interpretations of these findings are explored in the implications section. The 
MTDA discourages students from taking multiple courses during the same semester, 
and schools must file a request justifying why a student should be allowed to do so. 


Notes-1 


References 


Allensworth, E. M., & Easton, J. Q. (2007). What matters for staying on-track and graduating 
in Chicago public high schools: A close look at course grades, failures, and attendance in the 
freshman year (Research report). Chicago, IL: University of Chicago, Consortium on 
Chicago School Research. http://eric.ed.gov/licUED498350 

Allensworth, E. M., Gwynne, J. A., Pareja, A. S., Sebastian, J., & Stevens, W. D. (2014). 
Free to fail or on-track to college. Setting the stage for academic challenge: Classroom control 
and student support (Research brief). Chicago, IL: University of Chicago, Consortium 
on Chicago School Research. 

Alliance for Excellent Education. (2011). The high cost of high school dropouts: What the 
nation pays for inadequate high schools (Issue brief). Washington, DC: Author, http:// 
eric.ed.gov/?id=ED537542 

Archambault, L., Diamond, D., Brown, R., Cavanaugh, C., Coffey, M., Foures-Aalbu, D., 
et al. (2010). Research committee issues brief: An exploration of at-risk learners and online 
education. Vienna, VA: International Association for K-12 Online Learning, http:// 
eric.ed.gov/lkUED509620 

Bakia, M., Mislevy, J., Heying, E., Patton, C., Singleton, C., & Krumm, A. (2013). Support- 
ing K-12 students in online learning: A review of online Algebra 1 courses. Menlo Park, 
CA: SRI International. 

Benner, A. D., & Graham, S. (2009). The transition to high school as a developmental process 
among multiethnic urban youth. Child Development, 80(2), 356-376. http://eric.ed.gov/ 
?id=EJ837844 

Berliner, B., Barrat, V. X., Fong, A. B., & Shirk, P. B. (2008). Reenrollment of high school 
dropouts in a large, urban school district (Issues & Answers Report, REL 2008-56). 
Washington, DC: U.S. Department of Education, Institute of Education Sciences, 
National Center for Education Evaluation and Regional Assistance, Regional Educa- 
tional Laboratory West. http://eric.ed.gov/lkUED502274 

Blazer, C. (2009). Literature review: Virtual schools. Miami, FL: Miami-Dade County Public 
Schools, Research Services. http://eric.ed.gov/lkUED536253 

Bruce, M., Bridgeland, J. M., Fox, J. PI., & Balfanz, R. (2011). On track for success: The use 
of early warning indicator and intervention systems to build a grad nation. Washington, 
DC: Civic Enterprises. http://eric.ed.gov/lkUED526421 

Carver, P. R., & Lewis, L. (2011). Dropout prevention services and programs in public school 
districts: 2010-11 (First Look, NCES No. 2011037). Washington, DC: U.S. Department 
of Education, National Center for Education Statistics. http://eric.ed.gov/lkUED524175 

Day, J. C., & Newburger, E. C. (2002). The big payoff: Educational attainment and synthetic 
estimates of work-life earnings. Washington, DC: U.S. Department of Commerce, U. S. 
Census Bureau. http://eric.ed.gov/lkUED467533 


Ref-1 


Franco, M. S., & Patel, N. H. (2011). An interim report on a pilot credit recovery program 
in a large, suburban Midwestern high school. Education, 132(1), 15-27. 

Frazelle, S., & Nagel, A. (2015). A practitioner’s guide to implementing early warning systems 
(What’s Known Report, REL 2015-056). Washington, DC: U.S. Department of Edu- 
cation, Institute of Education Sciences, National Center for Education Evaluation and 
Regional Assistance, Regional Educational Laboratory Northwest, http://eric.ed.gov/ 
?id=ED552306 

Greaves, T. W., & Hayes, J. (2008). America’s digital schools 2008: The six trends to watch. 
Encinitas, CA: Greaves Group. 

Hagen, A. (2011, June 6). Juneau hosting rural education leaders in Montana (News release). 
Helena, MT: Montana Office of Public Instruction. Retrieved January 13, 2014, from 
http://opi.mt.gov/Media_Center/News_Updaters/NewsStories/2011-06-06_093520. 
html. 

Heckman, J. J., & LaFontaine, P. A. (2007). The American high school graduation rate: 
Trends and levels (NBER Working Paper No. 13670). Cambridge, MA: National Bureau 
of Economic Research. http://eric.ed.gov/lkUED501983 

Levin, H. M., Belfield, C., Muennig, P, & Rouse, C. (2007). The public returns to public 
educational investments in African-American males. Economics of Education Review, 
26(6), 699-708. http://eric.ed.gov/lkUEJ781067 

Mileaf, M., Paul, A., Rukobo, E., & Zyko, A. (2012). Informational brief: A look at credit 
recovery programs across the United States. Brooklyn, NY: RMC Research Corpo- 
ration, New York Comprehensive Center. Retrieved May 11, 2015, from http:// 
nycomprehensivecenter.org/2013/03/informational'brief'a'look-at'Credit'recovery 
-programs-across-the-united-states/. 

Miller, R. (2012). Montana Digital Academy: A report prepared for the Legislative Finance 
Committee. Helena, MT: Montana Legislature, Legislative Fiscal Division. Retrieved 
January 13, 2014, from http://leg.mt.gov/content/Publications/fiscal/interim/2012_ 
financemty_Nov/2012_ll_01_DM_Montana%20Digital%20Academy.pdf. 

Montana Digital Academy, (n.d.). What is the MTDA Connect Credit Recovery Program ? 
Retrieved May 1, 2015, from http://montanadigitalacademy.wikispaces.com/What+is+ 
the+MTDA+Connect+Credit+Recovery+Program%3F. 

Montana Office of Public Instruction, Graduation Matters Montana. (2015). 2015 gradu- 
ation and dropout report. Retrieved January 27, 2016, from http://www.opi.mt.gov/pdf/ 
Measurement/2015Graduation_Dropout.pdf. 

Muennig, P. (2005, October). Health returns to education interventions. Paper presented at 
the Social Costs of Inadequate Education Symposium, Columbia University, Teachers 
College, New York, NY. 


Ref-2 



Oliver, K., Osborne, J., Patel, R., & Kleiman, G. (2009). Issues surrounding the deployment 
of a new statewide virtual public school. Quarterly Review of Distance Education, 10(1), 
37-49. http://eric.ed.gov/lkNEJ864041 

Pettyjohn, T., Kennedy, K., & LaFrance, J. (2013). Supporting students in supplemental 
online learning for credit recovery. In R. McBride & M. Searson (Eds.), Proceedings of 
Society for Information Technology & Teacher Education International Conference 2013 
(pp. 4141-4146). Chesapeake, VA: Association for the Advancement of Computing in 
Education. 

Picciano, A. G., & Seaman, J. (2009). K—12 online learning: A 2008 follow-up of the survey 
of V.S. school district administrators. Newburyport, MA: Sloan Consortium. http://eric. 
ed.gov/?id=ED530104 

Picciano, A. G., Seaman, J., Shea, P, & Swan, K. (2012). Examining the extent and 
nature of online learning in American K-12 education: The research initiatives of the 
Alfred P. Sloan Foundation. Internet and Higher Education, 15(2), 127-135. http://eric. 
ed.gov/?id=EJ957652 

Rosenkranz, T., de la Torre, M., Stevens, W. D., & Allensworth, E. M. (2014). Free to fail 
or on-track to college: Why grades drop when students enter high school and what adults 
can do about it (Research brief). Chicago, IL: University of Chicago, Consortium on 
Chicago School Research. 

Rouse, C. E. (2005, October). Labor market consequences of an inadequate education. Paper 
presented at the Social Costs of Inadequate Education Symposium, Columbia Univen 
sity, Teachers College, New York, NY. 

Simmons, R. G., & Blyth, D. A. (1987). Moving into adolescence: The impact of pubertal 
change and school context. New York, NY: De Gruyter. 

Strange, M., Johnson, J., Showalter, D., & Klein, R. (2012). Why rural matters 2011-12: The 
condition of rural education in the 50 states. A report of the Rural School and Community 
Trust Policy Program. Arlington, VA: Rural School and Community Trust. http://eric. 
ed.gov/?id=ED528634 

Sum, A., Khatiwada, I., & McLaughlin, J. (with Palma, S.). (2009). The consequences of 
dropping out of high school: Joblessness and jailing for high school dropouts and the high cost 
for taxpayers. Boston, MA: Northeastern University, Center for Labor Market Studies. 

U.S. Bureau of Labor Statistics. (2015). Employment projections: Earnings and unemployment 
rates by educational attainment. Retrieved May 8, 2015, from http://www.bls.gov/emp/ 
ep_chart_001 .htm. 


Ref-3 



The Regional Educational Laboratory Program produces 7 types of reports 



Making Connections 

Studies of correlational relationships 


Making an Impact 

Studies of cause and effect 


What’s Happening 

Descriptions of policies, programs, implementation status, or data trends 


What’s Known 

Summaries of previous research 


Stated Briefly 

Summaries of research findings for specific audiences 


Applied Research Methods 

Research methods for educational settings 


Tools 

Help for planning, gathering, analyzing, or reporting data or research 



